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1.  Introduction.  We  arc  concerned  with  a  class  of'firoblcms  described  in  a  somewhat 

imprecise  way  as  follows.  Consider  a  linear  operator  of  the  form  L  +  V(x),  where 

L  is  the  generator  of  a  Markov  process  x]  imd  thc^potcntial^Vfx)  is  some 

vy  s<j\)  ^ 

real -valued  function  on  the  state  space  £  '  of  xl .  .We  arc  interested  in  probabili- 
stic  rj^uuisiwtatigns  for  solutions  <J>(s,x)  toNsWnackward  equation  \ 


(1.1) 


+  14  +  V(x)<j)  =  0,  s  <  T, 


with  data  <KT,x)  =  l(x)  at  a  final  time  T.  It  is  well  known  that,  under  suitable 


assumptions! 


(1.2) 


i 

4>(s,x)  =  Esx  {4>(xT)cxp  V(xt)dt) 


gives  such  a  representation.  For  instance,  if  xt  =  x  +  wt  -  'V  ’  wt  11  \ 

brownian  motion,  then  (1.2)  is  just  the  Fcynman-Kac  formula.  We  seek  a  different  \ 
kind  of  probabilistic  representation  for  J  ~  -log  f,  if  '(>(s,x)  is  a  positive 
solution  to  (1.1).  In  this  representation  the  generator  I.  is  replaced  by  another 
generator  L-  of  a  Markov  process  £;  (possibly  time  inhomogeneous.)  The  operator 

L—  is  chosen  to  solve  an  optimal  stochastic  control  problem  of  the  following  kind. 
The  logarithmic  transformation  l  =  -log/J)  changes  (1.1)  into  the  nonlinear  equation 


0.3) 


(1.4) 


cis +  II(I)  ■  v(x-)  =  wl,cro 


11(1)  =  -clL(c_i) 


The  function  II  is  concave.  For  a  fairly  wide  class  of  Markov  processes,  we  wish 
to  write  (1.3)  as  the  dynamic  programming  equation  associated  with  a  suitable  optimal 
stochastic  control  problem  for  Markov  processes.  The  stochastic  control  problem  is 
specified  by  giving:  (a)  a  suitable  control  space  U;  for  each  constant  control 
u  €  U,  the  generator  LU  of  a  Markov  process;  and  (c)  a  cost  function  k(x,u) 
associated  with  constant  control  u  and  state  See  [6,  Chap.  VI j.  It  is  rc- 
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% 

‘quir<Al  that 

(1.5)  Il(IJ(x)  =  nun  [LuI(x)  +  k(x,w)],  x  €  I  . 

u€U 

Then  (1.3)  becomes  a  dynamic  programming  equation: 

(1.6)  ~+  min  [LUI  +  k(x,u)  -  V(x)]  =  0. 

as  u€U 

Time  and  state  dependent  controls  u(s,x),  in  feedback  form,  with  values  in  the 
control  space  U  arc  allowed.  The  stochastic  control  problem  is  to  find  a  feedback 
u  minimizing 

fT 

(1.7)  J(s,x;u)  =  Esx  [k(5  ,ut)  -  V(5  )]dt  +  Y(5T)  , 

^  s 

where  K.  is  the  (controlled)  Markov  process  with  generator  L— ,  C  =  x,  and 

t.  5 

ut  =  u(t,St),  Y  =  -log  4>  . 


The  Verification  Theorem  of  optimal  stochastic  control  theory  [(>, 
if  I  is  a  "well  behaved"  solution  to  (1.3)  with  I(T,x)  =  Y(x), 
'■■'HinicaL  conditions  hold,  then 


p.l5‘.)j  asserts  that 
and 'if  certain  other 


)  I(s,x)  -  min  .l(s,x;u)  . 

u  .  “ 

Moreover,  an  optimal  feedback  control  >u(s,x)  is  found  by  minimizing  LuI(s,x)  +  k(x,u) 
over  the  control  space  U. 


In  this  paper  we  take  E  c  Rn,  a  subset  of  n-dimenslonal  euclidean  spjicc.  In  02 
we  review  the  case  when  xt  is  a  diffusion  process  on  R  .  For  nondegenerate  dif¬ 
fusions,  an  appropriate  stochastic  control  problem  is  immediately  suggested  by  the 
form  of  equation  (1.3).  In  §3  we  consider  jump  Markov  processes  xt>  and  associated 
stochastic  control  problems.  The  choice  of  an  appropriate  control  problem  is  less 
immediate  for  jump  processes  than  for  diffusions.  In  his  I’h.i).  thesis  S-.J  Shcu  [111;, 
uses  a  different  Control  formulation,  valid  for  a  wide  class  of  generators  L(§4). 

The  optimal  control  in  his  sense  leads  to  the  change  of  probability  measures  described 
in  (4.5).  In  §5  we  give  a  formal  derivation  indicating  why  stochastic  control  methods 

E 

can  be  used  to  obtain  asymptotic  estimates,  for  exit  probabilities  for  a  family  xt 
of  nearly  deterministic  jump  processes.  The  results  are  not  new  (sec  |T|[12|);  the  • 
interest  is  in  the  stochastic  control  method.  Rigorous  proofs  are  given  in  [11]  using 
such  methods. 


In  06  we  consider  briefly  the  Donsker-Varadhan  formula  for  the  dominant  eigenvalue 

hastic 

.outj.oa/ 


A^  of  L+V,  from  a  control  viewpoint.  For  nondcgcncratc  diffusions  the  stochastic 


control  representation  obtained  for  A^  is  the  same  as  Holland's  [9] . 


Availability  Codas 
[Avail  and/or 
Spaolal 


The  stochastic  control  representation  (1.8)  was  used  in  [3]  to  give  a  stochastic 
control  proof  of  results  of  Ventsel-Freidlin  type  for  some  large  deviations  problems 
for  nearly  deterministic  diffusions.  In  those  results  a(x)  is  replaced  by  ea(x), 
e  small.  In  [4]  the  logarithmic  transformation  was  used  to  obtain  stochastic 
representations  for  positive  solutions  to  the  heat  equation  with  a  potential  term, 
and  to  obtain  the  "classical  mechanical  limit."  In  [5]  [10]  the  same  logarithmic 
transformation  was  applied  to  solutions  to  the  pathwise  equation  of  nonlinear  filtering. 
Large  deviations  results  for  the  nonlinear  filter  problem  arc  obtained  by  Hi  jab 
[8]  elsewhere  in  this  volume. 

In  [7]  Hernandez- Lorma  obtained  similar  results  for  certain  degenerate  diffusions, 

for  which  the  matrix  (a..(x)),  i, j=l,--^-,r.i  <  n  is  positive  definite  and  a.  .(x)=0 

l  j  x  j. 

if  i  >  in  or  j  >  m. 

3.  Jump  processes.  To  motivate  our  choice  of  stochastic  control  problem,  let  us 
begin  with  a  simple  special  case  in  which  the  process  x^.  jumps  only  by  a  fixed 
increment  y  (as  for  example  for  a  Poisson  process.)  In  this  case  the  generator 
L  takes  the  form 


Lf(x)  =  a(x) [f (x+y)-f (x) ] . 


11(1) (x)  =  a(x) (1  -  exp  [L(x)  -  1 (x+y) J ) . 


From  \*1.'4) 

The  dual  function  to  the  convex  function  cr  is  u  -  u  log  u  (u  >  0) : 

(3.1)  c  =  max  [u  -  u  log  u  +  ur]  . 

u>0 

The  max  occurs  when  log  u  =  r.  Let 

(3.2)  LUI(x)  =  ua(x) [I(x+y)-I(x)] ,  u  >  0 

(3.3)  k(x,u)  =  a(x)(u  log  u  -  u+1). 

By  taking  r,  =  I(x)-I(x+y)  in  (3.1)  and  changing  signs  (to  replace  max  by  min), 
we  get  the  required  form  (1.5)  for  II(I).  In  this  special  case  the  control  u 
is  scalar,  with  u  >  0.  A  constant  control  u  changes  the  jumping  rate  from  a(x) 
to  tia(x).  A  feedback  control  u(s,x)  changes  the  rate  at  time  s  and  state  x  from 
a(x)  to  u(s,x)a(x).  If  ,T(s,x)  =  -log  f(s,x)  as  in  §1,  then  the  optimal  feedback 
control  is  u*(s,x)  =  <J>(s,x)“^<j>(s,x+y) . 

Let  us  now  consider  a  jump  process  xt  with  generator  of  the  form 

(3.4)  Lf(x)  =  a(x)  f  [f (x+y)  -  f (x)]-ir (x,dy) . 

Rn  . 

Here  f  6B(Rn),  the  space  of  bounded  Borci  Measurable  functions  on  R  .  We  assume 
that  a  6  B(Rn)  and  that  tr(x,*)  is  a  probability  measure  with  n(*>A)  Borci 
mensurable  for  each  Borci  set  A  and  1T (x, { 0} )  =  0.  Additional  conditions  on  a 
and  i'  need  to  be  imposed  later.  Motivated  by  the  special,  case  above,  we  control 
the  jumping  distribution.,  replacing  a(x)n(x,dy)  by  a(x)u(s,x;y)ir(dy) .  To. formalize 
this  idea,  we  introduce  the  control  space 

(3.5)  U  =  (u(*) :  u,u_1G  B(Rn) ,  u(y)  >  0  for  ail  y  G  It"  J. 

Suitable  I.u^  ^  ar.d  k(x,u(*))  nre  obtained  by  integrating  (3.2),  (3.3). with  respect 
to  ir(x,dy): 

(3.6)  LU^I'(x)  =  a(x)  f  n[I(x+y)-l  (x)  Ju(y)ii(x,dy) 

JRn 

(3.7)  k(x,u(* ))  =  a(x)  f  [u(y)log  u(y)-u(y)+l]ir(x,dy) . 

V 

We  get  as  in  equation  (1.5). 

(3.8)  11(1)  (x)  =  min  [LU^I(x)+k(x,u(*))J 

u(/)€U 

If  «Ks>x)  is  a  positive  solution  to  (1.1)  and  1  =  -logifi  ,  then  the  optimal  feed- 


back  control  is 


(3.9) 


* 


u  (s,x;*) 


TcsTxT" 


As  outlined  in  the  next  section,  it  is  sometimes  more  convenient  to  consider  instead 
a  related  control  problem.  In  particular,  the  formulation  in  §4  is  the  one  used  in 
[11]  to  give  control  method  proofs  of  the  results  on  the  exit  problem  mentioned  in 
§5.  '  . 

4.  The  Sheu  formulation.  In  [11]  another  kind  of  control  problem  is  considered.  Let 
L  be  a  bounded  linear  operator  on  C(Z),  the  space  of  continuous  bounded  functions 
on  Z  ,  such  that  L  obeys  a  positive  maximum  principle.  (In  particular,  L  may 
be  of  the  form  (3.4)  above.)  Tor  w  =  w(*)  a  positive  function  with  w,w  *  G  C(E), 
define  the  operator  Lw  by 

(4.1)  LWf  =  £  [LCwf)  -  £Lw]. 

In  addition,  define  (x)  by 

(4.2)  kw’=  Dv-(logw)  -  ±L(w). 

For  tinboundcd  L,  additional  restrictions  on  w  arc  needed  in  order  that  Lw  and  kw 
be  well  defined. 


From  the  duality  (3.1)  between  er  and  u  log  u  -  u,  it  is  not  difficult  to  show  [1J | 
that  for  I  €  C(I) 

(4.3)  Il(i)  =  min  [ Lw  I  +  K*]  . 

w 


The  minimum  is  attained  for  w  =  exp  (-1).  For  1,  the  generator  oF  a  jump  process, 
the  two  formulations  arc  related  by  ll*  =  I,—  ,  where  u_  is  the  (stationary)  feedback 
control  defined  by 


(4.4) 


u(x;y) 


w (x+y) 
"  w(x) 


Moreover,  Kw(x)  =  k(x,u(x;*))* 


In  Sheu's  formulation,-  the  control  problem  is  to  choose  w  (•)  for  s  <_  t  <  T 
minimize 


/(s,x;w)  =  nsx  {  fT[KWt-Kt)  -  VKt)]dt  +  m.T))  , 

s  ,w  * 


ic  ri  Mn-rl'mi  nrrvnqt-.  with  I. 


to 


ti'linro  r 


:ind  w  i  1 1i  r 


“  v . 


More 


I 

wo  assume  that  L  is  the  generator  of  a  Markov' process  xt  which  implies  in  particu¬ 
lar  L  1  =  1. 

Suppose  that  <t>  is  a  positive  solution  to  (1.1),  with  (s , • ) ,  '(’(s,*)"*  e  C(E) 
and  with  V  €C(I).  IVc  can  use  (4.3)  together  with  the  Verification  Theorem  in 
stochastic  control  to  conclude  that  I(s,x)  £  yf(s,x;wj  with  equality  when 

k 

wt  =  ^  Ci » • )  •  Thus  the  control  w  =  <j>(t,»)  is  optimal  in  this  sense.  'For  jump 
processes  this  agrees  with  (3.9),  according  to  (4.4). 


The  change  of  generator  from  L  to  L  =  L  z  corresponds  to  a  change  of  probability 
measure,  from  P  to  P,  as  folLows: 


(4.5) 


h  |f(xj<l>(x  ,.,)] 

*  -  ■ XjiiJ  ,  •'  5  i  C  i  T’  f  e  • 


This  is  seen  from  the  foliowing  argument.  The  denominator  of  the  right  side  is 
<j>(s,x).  hot 

iKs,x),=  hsx[f(xt)4>(xT)J  =  Hsx[f(xt)Kt,xt)]  . 

Since  <J>  and  ip  both  satisfy  (1.1)  with  V  =  0,  the  quotient  v  »  #  1  satisfies 

£■-  &-*%!•  -k  HM)  -vLil, 


(4  .  (i) 


l*v  ■  0.  S£t. 


with  v(t,x)  =  f(x)  as  required. 


The  author  wishes  to  thank  M.  Day  for  a  helpful  suggestion  related  to  (4.5). 


5 .  Asymptotic  estimates  for  exit  probabilities . 

Let  xj  be  a  family  of  Markov  processes,  s  <  t  <  T,  depending  on  a  small  parameter 

z  E  0 
e  >  0,  such  that  xfc  tends  (in  a  suitable  sense)  to  a  deterministic  limit  x^.  as 

G  ->  0.  Let  <|>  denote  the  probability  that  x  belongs  to  a  set  P  of  trajectories 

0  c 

which  docs  not  include  trajectories  "near"  x  .  Typically  <|>  is  exponentially 
small.  Its  asymptotic  rate  of  decay  to  0  can  bo  found  from  the  theory  of  large 
deviations  [l][12]p3].  In  the  exponent  a  constant  I1  appears,  which  is  the  mini¬ 
mum  of  a  certain  action  functional  over  a  set  of  smooth  paths. 


In  many  instances  these  asymptotic  estimates  can  also  be  obtained  by  introducing  a 
stochastic  control  problem  of  the  kind  indicated  in  previous  sections,  for  each  G  >  0 
|3|  1111.  With  this  method  a  (stochastic)  optimization  problem  .appears  for  each 


‘  e  > 


0,  not  just  in  the  limit  as  e  -*■•(). 


Let  us  consider  the  special . case  when  lP  is  an  exit  probability: 

+e(s,x)  =  Psx(Te  <  T), 

where  is  the  exit  time  of  x^  from  a  bounded,  open  set  D c  Rn,  and  where  x^.  €  1) 

for  s  <  t  <  T.  We  consider  nearly  deterministic  jump  processes,  as  follows.  Nearly 

deterministic  diffusions  were  considered  .in  [3]  [7].  following  Vcnt'cel  [12]  let  us 

rescale  the  jump  process  in  §3,  replacing  y  by  ey  and  a(x)  by  c  ^a(x)  to 

£ 

obtain  the  generator  for  xt: 


(5.1) 


b  f(x)  = 

£ 


=  e_1a(x)  [f 

nil 


[F(x+uy)-ffx)]  ir  (x ,  dy ) 


fix  xc  =  x.  for  s  <  t  <  T,  the  path  xC  tends  in  probability  as  e  ->■  0  (D-mctric) 


0 

to  xp  ,  where  x^  satisfies 

.  0 

(5.2)  =  a(xj)  [  yii(xj,dy),  s  <  t  <  T, 

J  R'i 

with  x°  =  x.  The  exit  probability  ♦t(s,x)  is  a  positive  solution  to 


(5.3) 


,  e 

+  i.  <r  =  o 

9s  c 


In  (-"Vr)  x  I).  The  logarithmic  transformation  T '  =  -i:  log  'I1  changes  (5.3)  into 


(5.4) 


*  e  lys-'f)  .  o. 


where  H£ ( T )  =  -c*Le(c  *) .  Then 


(5.5)  elle(e_1I)  =  a(x)  (1  -  cxnf1  |)n Cx,dy) 

l" 

for  I(x)  such  that  1,  I  are  continuous,  bounded 


liin  e  u  (e_iI)  =  Iin(x,  I  ), 
f^-0  u  x 


with  I  the  gradient  and 

A 


(5.6) 


II  (x,p)  =  a (x)  [  (1  -  c"p  *  y)n(x,dy) 

J  tl 


e  .  0 

This  suggests  (but  certainly  docs  not  prove)  that  1  tends  to  a  limit  I  as  ->■  0, 
where  satisfies  (perhaps  in  some  generalized  sense) 


V 


l 


(5.7) 


31° 


+  ll(x,I°)  =  0. 


Now  (5.7)  is  the  dynamic  programming  equation  for  the  deterministic  control  problem 
with  control  space  U  as  in  §2,  with  running  cost  k(£t,ut(*)) and  with  dynamics 

dT,  ./ 

(5.8)  —  »  l>(St,ut  (••)), 

b(x,u(*))  =  a(x)  |  y  u(y)ir(x,dy) . 

V1 


e  n 

Shcu  [11]  proved  that  indeed  I  -*■  I  as  e  -v  0  under  the  following  hypotheses: 

(,i)  a(»)  is  bounded,  positive,  and  l.ipschitz; 

(ii)  it (x,dy)  =  g(x,y)TrJt(dy)  with  ^  a  probability  measure,  ^({0})  =  0, 
g(*,y)  uniformly  Lipschitz,  and  0  <  c^  <  g(x,y)  <_  c2; 

(iii)  [  exp  (a|y|2)n  (dy)  <  °°  for  some  a  >  (); 

iRn 

(iv)  the  convex  hull  of  the  support  of  n.  contains  a  neighborhood  of  0  . 


Condition  (i-v)  insures  that  lln.(x,p)  is  the  dual  of  the  usual  "action  integrand"  A(S,5) 

^  #  II 

in  large  deviation  theory,  where  for  6  11 

(5.91  A(^,5)  =  min  [k(5,u(*))  =  i  =  b(5, u(*))J  • 

u0 


Then 

(5.10) 


A(^t,Ct)dt,  x  e  I). 


1  (s,x)  =  min 

1  S.  's 

The  minimum  is  taken  among  C  paths  K.-  with  =  x  such  that  ^ 
()1)  at  time  0  _<  T.  The  requirement  in  (5.10)  that  exit  from  I) 
is  suggested  by  the  boundary  condition  t  (T,x)  -  +  ro  for  x  €  I). 
in  the  limit  as  c  -►  0  to  an  infinite  penalty  for  failure  to  reach 


Thi 


first  reaches 
hv  t:  i  me  T 
s  corresponds 
<)D  by  time  T. 


G.  0 

In  both  [3]  and  ( 1 l j  the  stochastic  control  method  used  to  show  that  1  -►  I  depends 

on  comparison  arguments  involving  an  optimal  stochastic  control  process  when  e  >  0 
and  an  optimal  5^.  in  (5.10)  when  e=0. 


6.  The  dominant  eigenvalue.  In  [2]  Donsker  and  Varadhan  gave  a  variational  formula 
[(6.4)bclow]  for  the  dominant  eigenvalue  Xj  of  I,  +  V.  Another .derivation  of  this 
formula  is  given  in  [11]',  using  the  family  of  operators  l.w  mentioned  in  54. 

When  L  is  the  generator  of  a  nondegenerate  diffusion  process,  Holland  [9]  expressed 
z\  l  as  the  minimum  average  cost  per  unit  time  in  a  stochastic  control  problem.  !>ot  us 


I  ' 

'impose  strong  restrictions  on  L,  and  give  a  short  derivation  of  (6.4). 

Assume  that  L+V  has  a  positive  eigenfunction  corresponding  to  X  ^ :  (L+V)<J»j=X  ^ . 

Let  1^  =  -log  ^  j  *  ^leM 

(6.1)  '  -I1(I1)  +  V  =  . 

Assuming  that  there  is  a  stochastic  control  representation  (1.5)  for  Il(I) ,  equation 

(6.1)  becomes 

(6.2)  min  [LuI1(x)  +  k(x,u)]  -  V(x)  =  -X  . 

ueu  •  1  1 

P.quation  (6.2)  is  the  dynamic  programming  equation  for  the  following  average  cost  per 
unit  tine  control  problem.  We  admit  stationary  controls  u(*)  such  that  the  con¬ 
trolled  process  with  generator  L—  has  an  equilibrium  distribution  U  .  The  criterion 
to  be  minimized  is 

(6.3)  J(p,u)  =  f  Tk(x,u(x))  -  V(x)]dp(x) . 

(If  there  is  a  unique  equilibrium  distribution  p  =  p  —  then  reference  to  |j  on  the 
left  side  of  (6.3)  is  unnecessary.).  The  principle  of  optimality  states  that 
-Xj£  J(p,u)  with  equality  provided  u*(x)  gives  the  minimum  over  u  6  l)  of 

LUIj(x)  +  k (x , u) . 


Let  us  now  assume  that  1  is  compact,  that  the  generator  L  is  bounded  on  (!(£) 

and  V  £  C(E).  As  in  [2j  for  any  probability  measure  p  on  £  let 


_^(p)  =  sup 
I 


•  H(T)dji  =  -inf  f  ^  dp 
<J>>  0  J  l  ! 


) 


where 

(6.4) 


T,  1*  €C(E). 


The  Donskcr-Varadhan 

X  =  sup  [ 
p 


formula  is 

VdP  -j?(v )]. 

n 


Let 


The  function  P 
can  find  I ^ ,  p 


P(I,U) 


-11(1)  +  V]dp  . 


is  convex  in  I  and  linear  in  p  .  formula  (6.4)  will  follow  if  wc 
with  the  saddle  point  property: 


(6.5) 


PUpM)  lxl  1  PCI.Uj)  for  all  I,  p. 


(This  idea  was  known  to  Donskcr  and  Vorndhan  a  long  time  ago,  and  figures  in  their 


Prom  (6.1)  we  have  in  fact  P(I  ,P)  =  X  For  all  probability  measures  P  on  X  . 

11  jj3 4 5 6 7Ar 

To  get  the  right  band  inequality,  choose-  u*  as  above  and  assume  that  l,“  is 
bounded  on  C(X).  The  corresponding  Markov  process  has  an  equilibrium  distri¬ 
bution  Pj,  and 


(6.6) 


^(L^Dd^  =  0, 


for  al 1 


i  e  C0. 


(If  L--  is  unbounded  we  heed  to  assume  the  existence  of  |i,,  and  to  restrict 

u*  *  1 

I  to  the  domain  of  L^-  ) .  By  taking  u  =  u  (x)  in  (1.5)  we  have  for  1  'v.  C(X) 

* 


L-  I  +  k(x,u*)  -  V  >  H(I)  -  V. 


By  integrating  both  sides  with  respect  to  P  , 

=  J(brii*)  1  1  P(i,up, 

as  required. 


In  order  to  derive  (6.4)  in  this  way  wo  had  to  .impose  unnecessarily  restrictive 
hypotheses.  In  particular,  we  assumed  that  X^  is  a  dominant  eigenvalue  in  the 
strict  sense  that  (I.  +  V)*)^  =  X^,  with  ^  >  0*  Actually,  (6.4)  holds  if  L  is 
the  generator  of  a  strongly  continuous,  nonnegative  semigroup  T^  on  C(X),  such 
that  Tj.1  =  1*  L  has  domain  dense  in  C(X),  and  L  satisfies  the  maximum  principle 

[2].  With  such  assumptions  X^  is  a  dominant  eigenvalue  in  the  sense  that  the 
spectrum  of  1,  +  V  is  contained  in  {z:  lie  z  <_  Xf}  and  X}  -  (I,  +  V)  does  not 
have  an  inverse. 
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